doc: design decoupled coordination - #38696

Define catalog authority for maintained lifecycle and separate it from
request-scoped query execution. Record the ownership and readability
boundaries needed for independent clients, while leaving implementation
mechanisms open and full multi-adapter operation outside this deliverable.

Include an append-only implementation log for work across sessions.

Make sink export installation an implication of the committed catalog diff,
not a sequencer-side lifecycle call. Create exports after source and table
inputs, then initialize their default read policies. Existing read holds
protect inputs through installation, and SQL success still waits for it.

Extend sink compatibility coverage with no-op creation and rename operations.
Independent review found no blocking issue. CI Clippy and formatting passed
before a documentation push superseded the run. Runtime validation remains
pending, and local builds were blocked by missing artifacts and tools.

This is an incremental ownership migration. Cluster-side catalog following,
MV/index creation implications, and direct committed-update tests remain open.

Separate index reconstruction from bootstrap orchestration so committed
catalog implications can reuse it without inheriting bootstrap timestamp
selection or installation. Keep cache validation, dependency ordering,
and CREATE INDEX behavior unchanged.

This is preparatory work, not index lifecycle decoupling. Runtime cache
reads and committed index installation remain to be implemented.

Validation: adapter cargo check and Rust formatting passed. Independent
review found no issues. Full formatting and lint were blocked by local
tooling and dependency build failures. Runtime tests and PR CI are pending.

Make index-add implications acquire compatible cached plans or reconstruct
from catalog definitions, then install after cluster and storage creation.
Retain precommit optimization and success-only session notices, without a
sequencer installation closure. Reuse the existing read-hold handover.

Expose best-effort expression-cache reads through its task and validate
cached dependencies against committed state and available compute imports.
Filter session notices for optimizer dependencies dropped during planning.

Extend cache-handle tests and add cache-disabled SQL restart coverage for
index use, EXPLAIN, notices, and drop cleanup. Same-batch external catalog
application remains a test gap until a production subscriber exists.

Adapter and cache-test compilation, Rust and Python formatting, and Ruff
passed. Full local formatting and lint are blocked by missing tools.
Runtime validation is delegated to the draft PR CI and remains pending.

Register materialized view storage collections from committed implications
before dependent sinks and indexes acquire read holds. Share descriptor
construction with bootstrap to preserve initial visibility and replacement
shard ownership. Apply read policies through the deferred batch.

Keep compute installation and write enablement in the sequencer for this
incremental step. Extend the existing MV test with a dependent index.

Adapter cargo check and Rust/Python formatting pass. Independent review
found no issue. Full local formatting and lint are blocked by missing
tools and a doctest dependency build. Regular PR CI is pending.

Reuse bootstrap MV optimization and notice rendering as a catalog-based
planning boundary for compute implications. Preserve cache policy, SQL
typing, item versions, and timestamp selection. Compute installation and
input read protection remain unchanged.

Extend cache-disabled restart coverage to MV results, stored EXPLAIN
plans, and continued maintenance after restart.

Validation: adapter cargo check and Rust/Python formatting passed.
Independent review found no issue. Full formatting and lint are blocked
by missing tools and an OpenSSL dependency build. Runtime validation
and regular PR CI are pending.

Keep a REFRESH AT CREATION materialized view unexecuted across restart,
change its input after creation, and verify reconstruction uses the
committed refresh timestamp rather than current input state. Extend the
existing cache-disabled restart workflow without a test-only coordinator
API.

Record the input-protection obstacle to moving runtime MV installation.
Broader creator holds can change historical readability and do not cover
all access paths exposed by catalog changes.

Validation: Python formatting and Ruff passed. Full local formatting and
lint are blocked by missing tools and an OpenSSL dependency build. Runtime
coverage is pending regular PR CI.

Record explicit catalog bounds as the compaction authority for maintained
collections. Define the catalog transaction, application, and recovery
boundaries while leaving representation, granularity, and publication
cadence to implementation and measurement.

Document delegated advancement as an alternative and explain the tradeoff
between ongoing catalog publication and owner-backed DDL admission. Preserve
the exploration log and append the agreed decision.

Documentation checks passed, including preservation of earlier log entries.
Full formatting and lint remain blocked by missing tools and the Python
doctest OpenSSL dependency build. No runtime behavior changes.

…y tests

Strip validation churn from the design's implementation log: pending CI,
passed formatting checks, local tooling gaps, and review outcomes are all
reconstructible from the PR and buried the findings and decisions. Update the
session prompt so future handoffs record only findings, decisions, open
questions, and the next step, and note which local checks work.

Reduce the logical_collection_inputs tests to the promises that matter: name
resolution survives planning-time elimination, raw-HIR reads count, indexes do
not affect traversal, and upstream MV outputs are leaves. The dropped version
test mutated catalog maps directly, and the replacement-MV case cannot run in
the debug catalog because it has no shard metadata for the target.

Focus implementation on observable recovery, lifecycle, and query-client
outcomes in environments initialized under the new protection rules.
Defer existing-environment conversion without weakening its result
preservation contract.

Direct implementers to the active milestone and retain rejected
alternatives as decision context rather than pending work.

Account for committed compaction permission independently of retention
policies and execution holds. Preserve it during initialization, shared
shard version changes, and drops, and apply monotonic advances in batches.
Recovery uses protected readability rather than policy frontiers, and
initialization stops if a fencing compare fails.

This is the storage consumer boundary for decoupled coordination. Durable
catalog production, maintained admission, and MV recovery wiring remain
unimplemented, so SQL-created collections do not yet use these bounds.

Five focused storage tests cover persist advancement, recovery, aliases,
and drops. Recovery regressions were verified with the fixes removed.
Formatting, cheap lint checks, and all-target checks through adapter and
compute-client pass locally. Broad validation remains with draft PR CI.

Include logical collection inputs in MV refresh preparation and creation timestamp selection, even when planning or optimization removes their reads. Preserve early holds and acquire missing planner-introduced inputs before installation, rejecting explicit refreshes that those inputs cannot support.

Carry query-scoped name references separately from statement dependencies so replacement targets do not constrain input readability. Add SQL coverage for eliminated inputs, automatic timestamps, refresh-at-creation, replacement targets, and implicit SQL-function reads.

This prepares catalog-backed recovery protection but does not implement durable bounds or maintained requirements. Admission currently uses held readability.

Validation: bin/fmt, cargo check for mz-adapter and mz-sqllogictest with all targets, and selected Cargo, formatting, and Python-doc lint checks passed. SQL execution and broader validation are delegated to draft PR CI.

Accept committed removal of collection metadata during creation without
releasing the installed compaction bound. Catalog implications create
collections before applying drops, so requiring the removed bound in the
committed snapshot rejects valid mixed batches. Keep requiring bounds for
collections that remain cataloged and retain shared-shard protection until
explicit drop application.

Extend the version/drop boundary test to cover create-before-drop and
rejection of missing bounds for still-cataloged collections. The regression
failed before the fix. Targeted storage-bound tests, cargo check, formatting,
and selected repository lint checks passed. Regular PR CI remains pending.

This unblocks catalog-bound delivery but does not add its durable producer.

Persist storage-backed compaction bounds separately from maintained logical
read requirements. Validate final transaction state before batch extraction
and return admission failures before the adapter fatal commit boundary.
Deliver committed bounds through catalog implications and preserve collection
lifetimes in dry runs and drops.

This connects catalog authority to storage enforcement without enabling SQL
production of the records. Creation, recovery, and progress publication remain
work toward the first protected MV.

Add durable reopen and admission coverage, plus adapter coverage for committed
updates and staged collection lifetimes.

Keep valid client protection independent of the components enforcing
compaction, while query execution remains ephemeral. Record the volatile
alternative and its failure tradeoff without prescribing a component,
schema, or lease mechanism.

Add independent-client recovery evidence to milestone 3 and retain the
protected-MV production path as the immediate integration target.

Allow any component to apply committed monotonic permission without
exclusive ownership. Keep coordination at authorization, read admission,
and reclamation boundaries, with collection identity and all valid read
protection preserved.

Record concurrent and delayed application as milestone evidence without
removing fencing from controllers that still rely on private accounting.

Commit logical MV requirements and collection compaction permission before
installation, including fresh builtin collections. Advance requirements
from durable output progress and publish compatible bounds in batches,
while retaining valid execution and creation holds.

Recover using actual protected readability, co-register shared-shard
aliases, and preserve leased readability during read-only handover. Retry
finite persist since delivery through rate limits and primary changes.
Separate DDL conflict tracking from planning-cache freshness so background
protection and cadence controls preserve existing SQL behavior.

Enable the path at fresh-environment initialization, off by default in
production and on in tests. Existing-environment conversion and builtin
schema migration remain outside milestone 1.

Add a production-path demonstration of optimized-away input protection,
actual compaction, historical admission, uncached pending-refresh recovery,
and progress-driven history release. Keep catalog consistency comparisons
complete by quiescing publication while acquiring their snapshots.

Validation: formatting and targeted compilation checks pass. The milestone
demonstration passes in draft-PR CI. See the PR for broad check outcomes.

Record the architectural review priorities in the implementer prompt:
publication scaling, consistency-check coupling, and remaining ownership
dependencies. Keep the design unchanged and retire resolved steering
rather than accumulating a permanent checklist.

Keep milestone 1 active beyond the protected-MV checkpoint. Cover source
and sink recovery requirements and compute compaction before moving to
independent lifecycle execution.

Retain catalog-local authorization and its traffic tradeoff without
prescribing duplicate records or stronger historical recovery guarantees.

Keep builtin schema migration and protected-environment version upgrades
outside the fresh-environment decoupling deliverable, alongside existing
environment conversion. Demonstrate the ownership model before tackling
those follow-on concerns.

Preserve fresh builtin initialization, same-version recovery, and
ownership handover within the milestones.

A fresh index has no bound until first publication, and recovery installs at
the least readable frontier, capped by permission only where permission is at
or above readability. When it is below, nothing durable depends on the gap
because durable protection of an index protects its inputs, so the index is
replaced and its bound follows through publication. Installation never waits
for a catalog write. Record that client requirements on indexes protect the
index's inputs, which is what makes this safe under milestone 3.

Agree to move the consistency checker's reconstruction behind the existing
environmentd diagnostic check and remove the dedicated command, header, and
frozen catalog mode. Merge the paused steering items accordingly.

…ication

Publish index permission from installed readability, without birth records,
installation-time catalog writes, or read-only waits. Preserve feasible
saved permission and replace indexes at readability when it is not feasible.

Track changed controller frontiers and maintained input requirements.
Validate affected durable records using structurally shared projections,
and retire sparse bounds from final lifetime changes.

Reconstruct independent durable snapshots through the existing diagnostic
endpoint off the coordinator thread. Preserve ordinary invariant checks and
legacy compatibility without a frozen mode or dedicated snapshot command.

Add controlled recovery and publication-cost workflows, held-readability
coverage, and bounded-kind counters for committed catalog row payload.
Keep experiment dispatch out of the regular pipeline.

The large identical-index stress case exposes quadratic notice and dependency
work plus repeated uncached planning. Its scope split remains unresolved.
Milestone 1 is not declared complete.

Compare 100 and 1,000 generated objects while retaining the shared-view
index topology and diagnostics. Keep larger cohorts available explicitly,
and defer full-scale performance work beyond this milestone.

…dence

Include the generated inventory entries for committed catalog row counters.
Record the maintained-lifecycle handoff and keep experiment-only dispatch
out of the regular pipeline. The production measurement workflows retain
100/1,000-object defaults and explicit larger-cohort support.

Derive complete MV recovery requirements at the catalog transaction boundary, preserving query references eliminated by optimization and distinguishing replacement targets from reads. Validate surviving birth promises and writer identities against final transaction state.

Prepare storage metadata from final catalog membership rather than locally installed collections. Preserve shared-shard aliases and orphan reclamation without authorizing physical finalization.

These are prerequisites for independent lifecycle following. Compute installation and controller servicing remain adapter-owned.

MV replacement changes the catalog item ID but retains storage collection aliases. Pin collection versions in stored references independently of table schema evolution, and preserve explicit GlobalIds in session-catalog lookups.

Downstream recovery requirements then continue to match reconstructed SQL without treating shared shard identities as interchangeable permission.